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DETAILED ACTION 



Introduction 



I . This communication is in response to the Applicant's Response mailed on June 

I I , 2004. Claims 1-36 of the application are pending. This office action is made final. 



Claim Rejections - 35 USC § 103 



2. The following is a quotation of 35 U.S.C. 103(a) which forms the basis for all 
obviousness rejections set forth in this Office action: 

(a) A patent may not be obtained though the invention is not identically disclosed or described as set 
forth in section 102 of this title, if the differences between the subject matter sought to be patented and 
the prior art are such that the subject matter as a whole would have been obvious at the time the 
invention was made to a person having ordinary skill in the art to which said subject matter pertains. 



3. The factual inquiries set forth in Graham v. John Deere Co., 383 U.S. 1, 148 USPQ 459 
(1966), that are applied for establishing a background for determining obviousness under 35 
U.S.C. 103(a) are summarized as follows: 

1 . Determining the scope and contents of the prior art. 

2. Ascertaining the differences between the prior art and the claims at issue. 

3. Resolving the level of ordinary skill in the pertinent art. 

4. Considering objective evidence present in the application indicating obviousness 
or nonobviousness. 
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4. Claims 1, 3-10, 12, 13, 15-22, 24, 25, 27-34 and 36 are rejected under 35 U.S.C. 103(a) 
as being unpatentable over Viniotis et al. (VI) ("Linear programming . . . Queueing systems", 
IEEE, 1988) in view of Schneider et al. (SC) ("Stochastic Production scheduling . . . demand 
forecasts", IEEE, 1998). 

4. 1 VI teaches Linear programming as a technique for optimization of queuing systems. 
Specifically, as per Claim 13, VI teaches solving stochastic control problems of linear systems in 
high dimensions (Page 652, CLl, Para 1; Page 653, CL2, Para 3); comprising: 

modeUng a structured Markov Decision Process (MDP) (Page 652, CLl, Para 4; Page 
652, CL2, Para 6), wherein a state space for the MDP is a polyhedron in a Euclidean space (Page 
654, CL2, Lemma 2); 

one or more actions that are feasible in a state of the state space are linearly constrained 
with respect to the state (Page 653, CLl, Para 1 and Para 2; Page 652, CL2, Para 8); and 

building a value function for the state using representations that facilitate the computation 
of approximately optimal actions at any given state by linear programming (Page 653, CLl, Para 
9 to Page 654, CLl, Para 4; Page 652, CL2, Para 8). 

VI does not expressly teach a computerized apparatus for solving stochastic control 
problems of linear systems in high dimensions comprising a computer. SC teaches a 
computerized apparatus for solving stochastic control problems of linear systems in high 
dimensions comprising a computer (Page 2726, CLl, Para 3 and 4), as that allows the solution of 
stochastic control problems of linear systems in high dimensions run faster and allows the user to 
generate the results with varying data (Page 2726, CLl, Para 3). It would have been obvious to 
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one of ordinary skill in the art at the time of Applicant's invention to combine the method of VI 
with the apparatus of SC that included a computerized apparatus for solving stochastic control 
problems of linear systems in high dimensions comprising a computer type, as that would allow 
the solution of stochastic control problems of linear systems in high dimensions run faster and 
allow the user to generate the results with varying data. 

VI does not expressly teach logic performed by the computer, for modeling a structured 
Markov Decision Process (MDP). SC teaches logic p erfor med by thej:omputer, for modeling a 
structured Markov Decision Process (MDP) (Page 2726, CLl, Para 3 and 4), as that allows the 
solution of stochastic control problems of linear systems in high dimensions run faster and 
allows the user to generate the results with varying data (Page 2726, CLl, Para 3). It would have 
been obvious to one of ordinary skill in the art at the time of Applicant's invention to combine 
the method of VI with the apparatus of SC that included logic performed by the computer, for 
modeling a structured Markov Decision Process (MDP), as that would allov^ the solution of 
stochastic control problems of linear systems in high dimensions run faster and allow the user to 
generate the results with varying data. 

VI does not expressly teach logic performed by the computer, for building a value 
function for the state using representations that facilitate the computation of approximately 
optimal actions at any given state by linear programming. SC teaches logic performed by the 
computer, for building a value function for the state using representations that facilitate the 
computation of approximately optimal actions at any given state by linear programming (Page 
2726, CLl, Para 3 and 4), as that allows the solution of stochastic control problems of linear 
systems in high dimensions run faster and allows the user to generate the results with varying 
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data (Page 2726, CLl, Para 3). It would have been obvious to one of ordinary skill in the art at 
the time of Apphcant's invention to combine the method of VI with the apparatus of SC that 
included logic performed by the computer, for building a value function for the state using 
representations that facilitate the computation of approximately optimal actions at any given state 
by linear programming, as that would allow the solution of stochastic control problems of Unear 
systems in high dimensions run faster and allow the user to generate the resuUs with varying 

data. 

VI does not expressly teach logic performed by the computer, for building one or more 
approximations from above and from below to a value function for the state using 
representations. SC teaches logic performed by the computer, for building one or more 
approximations from above and from below to a value function for the state using 
representations (Page 2722, CLl, Para 2; Page 2724, CL2, Para 6), as value function 
approximation is an effective technique for both deterministic and noisy scenarios (Page 2722, 
CLl, Para 2); and approximation allows solving large scale MDPs (Page 2722, CL2, Para 2). It 
would have been obvious to one of ordinary skill in the art at the time of Applicant's invention to 
combine the method of VI with the apparatus of SC that included logic performed by the 
computer, for building one or more approximations from above and from below^ to a value 
function for the state using representations, as value function approximation would be an 
effective technique for both deterministic and noisy scenarios and approximation allows solving 
large scale MDPs. 



Application/Control Number: 09/607,5 1 3 Page 6 

Art Unit: 2123 

Per Claim 15: VI teaches that the action space and the state space are continuous and 
related to each other through a system of linear constraints (Page 652, CL2, Para 7; Page 653, 
CLl,Paral). 

4.2 As per Claim 16, VI and SC teach the apparatus of Claim 13. VI teaches that the value 
function is convex (Page 652, CLl, Para 4). 

VI doe^ not expressly teach that the logic further comprises eff icient ly learning the value 
function in advance and representing the value function in a way that allows for real-time choice 
of actions based thereon. SC teaches that the logic further comprises efficiently leaming the 
value function in advance and representing the value function in a way that allows for real-time 
choice of actions based thereon (Page 2722, CLl, Para 2), as that allows producing a time- 
dependent action policy specifically built to match the current demand forecasts and other 
system states (Page 2724, CL2, Para 4). It would have been obvious to one of ordinary skill in 
the art at the time of Applicant's invention to combine the method of VI with the apparatus of 
SC that included the logic further comprising efficiently leaming the value function in advance 
and representing the value function in a way that allows for real-time choice of actions based 
thereon, as that would allow producing a time-dependent action policy specifically built to match 
the current demand forecasts and other system states. 

4.3 As per Claim 17, VI and SC teach the apparatus of Claim 13. VI teaches that the linear 
function is represented by piecewise linear and convex functions (Page 652, CLl, Para 4; Page 
654, CLl, Lemma 1). 
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VI does not expressly teach the linear function is approximated both from above and 
from below by piecewise linear and convex ftinctions. SC teaches the linear function is 
approximated both from above and from below (Page 2722, CLl, Para 2; Page 2724, CL2, Para 
6), as value ftmction approximation is an effective technique for both deterministic and noisy 
scenarios (Page 2722, CLl, Para 2); and approximation allows solving large scale MDPs (Page 
2722, CL2, Para 2). It would have been obvious to one of ordinary skill in the art at the time of 
Applicant's invention to combine the method of VI with the a ppa ratus of SC that included the 
linear ftinction being approximated both from above and from below, as value ftmction 
approximation would be an effective technique for both deterministic and noisy scenarios and 
approximation allows solving large scale MDPs. 

Per Claim 18: VI teaches that the domains of linearity of the piecewise linear and convex 
ftinctions are not stored explicitly, but rather are encoded through a linear programming 
formulation (Page 652, CLl, Para 4; Page 654, CLl, Lemma 1; Page 653, CLl, Para 9 to Page 
654, CLl, Para 4). 

4.4 As per Claim 19, VI and SC teach the apparatus of Claim 17. VI teaches that the 
domains of linearity of the piecewise linear and convex ftinctions allow the ftinctions to be 
optimized and updated (Page 652, CLl, Para 4; Page 654, CLl, Lemma 1; Page 654, CLl, Para 
6 to Para 8). 

VI does not expressly teach that the domains of linearity of the piecewise linear and 
convex ftinctions allow the ftinctions to be optimized and updated in real-time. SC teaches that 
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the domains of linearity of the functions allow the functions to be optimized and updated in 
real-time (Page 2722, CLl, Para 2; Page 2724, CL2, Para 6), as that allows producing a time- 
dependent action policy specifically built to match the current demand forecasts and other 
system states (Page 2724, CL2, Para 4). It would have been obvious to one of ordinary skill in 
the art at the time of Applicant's invention to combine the method of VI with the apparatus of 
SC that included the domains of linearity of the functions allowing the functions to be optimized 
and updated in real-time, as that would allow produci n g a tim e-de p end e nt action p olicy 
specifically built to match the current demand forecasts and other system states. 

4.5 As per Claim 20, VI and SC teach the apparatus of Claim 13. VI does not expressly 
teach that the value function can be efficiently approximated both from above and fi-om below. 
SC teaches that the value function can be efficiently approximated both fi*om above and from 
below (Page 2722, CLl, Para 2; Page 2724, CL2, Para 6), as value function approximation is an 
effective technique for both deterministic and noisy scenarios (Page 2722, CLl, Para 2); and 
approximation allows solving large scale MDPs (Page 2722, CL2, Para 2). It would have been 
obvious to one of ordinary skill in the art at the time of AppUcant's invention to combine the 
method of VI with the apparatus of SC that included the value function being efficiently 
approximated both from above and from below, as value function approximation would be an 
effective technique for both deterministic and noisy scenarios and approximation allows solving 
large scale MDPs. 
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4.6 As per Claim 21, VI and SC teach the apparatus of Claim 1 3. VI does not expressly 
teach that the approximations can be repeatedly refined, SC teaches that the approximations can 
be repeatedly refined (Page 2725, CL2, Para 4), as that allows producing a time-dependent action 
policy specifically built to match the current demand forecasts and other system states (Page 
2724, CL2, Para 4). It would have been obvious to one of ordinary skill in the art at the time of 
Applicant's invention to combine the method of VI with the apparatus of SC that included the 
approximations^being repeatedly refined, as that would allow producing a time-dependent action 
policy specifically built to match the current demand forecasts and other system states. 

4.7 As per Claim 22, VI and SC teach the apparatus of Claim 13. VI does not expressly 
teach that the value fianction can be efficiently approximated firom above based on knowledge of 
upper bounds on the function at each member of a selected set of states. SC teaches that the 
value fiinction can be efficiently approximated from above based on knowledge of upper bounds 
on the fiinction at each member of a selected set of states (Page 2722, CLl, Para 2; Page 2724, 
CL2, Para 6; Page 2725, CL2, Para 3), as value fiinction approximation is an effective technique 
for both deterministic and noisy scenarios (Page 2722, CLl, Para 2); and approximation allows 
solving large scale MDPs (Page 2722, CL2, Para 2). It would have been obvious to one of 
ordinary skill in the art at the time of AppUcant's invention to combine the method of VI with 
the apparatus of SC that included the value function being efficiently approximated from above 
based on knowledge of upper bounds on the fiinction at each member of a selected set of states, 
as value fiinction approximation would be an effective technique for both deterministic and noisy 
scenarios and approximation allows solving large scale MDPs. 
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4.8 As per Claim 24, VI and SC teach the apparatus of Claim 1 3 . VI does not expressly 
teach that the value ftmction can be approximated successively. SC teaches that the value 
function can be approximated successively (Page 2725, CL2, Para 4), as that allows producing a 
time-dependent action policy specifically built to match the current demand forecasts and other 
system states (Page 2724, CL2, Para 4). It would have been obvious to one of ordinary skill in 
the art at the time of AppUcant's in ve ntion to combine the method of VI with the apparatus of 
SC that included the value function being approximated successively, as that would allow 
producing a time-dependent action policy specifically buiU to match the current demand 
forecasts and other system states. 

4.9 As per Claims 1, 3-10, 12, 25, 27-34 and 36, these are rejected based on the same 
reasoning as Claims 13, 15-22, 24, supra. Claims 1, 3-10, 12, 25, 27-34 and 36 are computer 
based method and article of manufacture embodying computer logic claims reciting the same 
limitations as Claims 13, 15-22 and 24, as taught throughout by VI and SC. 

5. Claims 2, 14 and 26 are rejected under 35 U.S.C. 103(a) as being unpatentable over 
Viniotis et al. (VI) ("Linear programming ... Queueing systems", IEEE, 1988) in view of 
Schneider et al, (SC) ("Stochasfic Production scheduling ... demand forecasts", IEEE, 1998), 
and further in view of Dangat et al. (DA) (U.S. Patent 5,971,585). 
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5.1 As per Claim 14, VI and SC teach the apparatus of Claim 13. VI does not expressly 
teach that the MDP comprises a supply chain planning process. DA teaches that the MDP 
comprises a supply chain planning process (Abstract; CLl, L7-21; CL6, L5-9), as that provides a 
computerized decision support tool to generate a best can do (BCD) match between existing 
assets and demands across multiple manufacturing and other facilities based on process flows 
and business policies (Abstract). It would have been obvious to one of ordinary skill in the art at 
the time of Applicant's invention to combine the method of VI wit h th e apparatus of DA that 
included the MDP comprising a supply chain planning process, as that would provide a 
computerized decision support tool to generate a best can do (BCD) match between existing 
assets and demands across multiple manufacturing and other facilities based on process flows 
and business policies. 

5.2 As per Claims 2 and 26, these are rejected based on the same reasoning as Claim 14, 
supra. Claims 2 and 26 are computer based method and article of manufacture embodying 
computer logic claims reciting the same limitations as Claim 14, as taught throughout by VI, SC 
and DA. 

6. Claims 1 1, 23 and 35 are rejected under 35 U.S.C. 103(a) as being unpatentable over 
Viniotis et al. (VI) ("Linear programming . . . Queueing systems", IEEE, 1 988) in view of 
Schneider et al. (SC) ("Stochastic Production scheduling ... demand forecasts", IEEE, 1998), 
and further in view of Hedlund et al. (HE) ("Optimal control of hybrid systems", IEEE 1999). 
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6.1 As per Claim 23, VI and SC teach the apparatus of Claim 13. VI does not expressly 
teach that the value function can be efficiently approximated from below based on linear 
functions that lie below the convex value function. HE teaches that the value function can be 
efficiently approximated from below based on linear functions that lie below the convex value 
function (Page 3972, CLl, Para 1; Page 3973, CLl, Para 3; Page 3977, CLl, Para 1), as that 
would provide a lower bound on the optimal cost in terms of linear programming (Page 3972, 
CL2, Para 2). It would have been obvious to one of ordin ary s kill in the ar^at the time of 
Applicant's invention to combine the method of VI with the apparatus of DA that included the 
value function being efficiently approximated from below based on linear functions that lied 
below the convex value function, as that would provide a lower bound on the optimal cost in 
terms of linear programming. 

6.2 As per Claims 1 1 and 35, these are rejected based on the same reasoning as Claim 23, 
supra. Claims 1 1 and 35 are computer based method and article of manufacture embodying 
computer logic claims reciting the same limitations as Claim 23, as taught throughout by VI, SC 
and HE, 

Response to Arguments 

7. Applicant's arguments filed on June 11, 2004 have been fully considered. Applicant's 
arguments filed on June 11, 2004 under 35 U.S.C. 
103 (a) are not persuasive. 
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7.1 As per the applicant's argument that "the Office Action asserts that Viniotis teaches a 
state space for the MDP is a polyhedron in a EucHdean space, at Page 654, CL2, Lemma 2; 
however, at the indicated location, Viniotis merely states ... in Viniotis, A is a constraint matrix, 
not a state space; moreover, Viniotis does not refer to a polyhedron in Euclidean space", the 
examiner respectfully disagrees. 

Viniotis states that the solution to the Linear Programming problem is an extreme point 
(Page 654, CL4, Para 6); extreme points form a polyhedron (Page 654, CL4, Para 6). One of 
ordinary skill in the art would have known that such polyhedron existed in the Euclidean space (a 
multi-dimensional space). The constraints of the linear program are lines in the multi- 
dimensional space forming the edges of the polyhedron. The constraints are defined by the 
states. Therefore, the state space of the linear program exists in an Euclidean space and is defined 
by a polyhedron. It is well known that a Markov decision Problem (MDP) is equivalent to a 
Linear Program; a MDP problem can be generally formulated as an equivalent Linear Program 
(Page 652, CLl, Para 4). Therefore, one of ordinary skill in the art would conclude that a state 
space for the MDP is a polyhedron in a Euclidean space. 

7.2 As per the applicant's argument that "the Office Action asserts that Viniotis teaches one 
or more actions that are feasible in a state of the state space are linearly constrained with respect 
to the state at Page 653, CLl, Para 1 and Para 2; Page 652, CL2, Para 7; however, at the 
indicated locations, Viniotis merely states . . .it can be seen that Viniotis teaches only that a linear 
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cost functional that involves the state is linear; however, these portions in Viniotis do not teach 
or suggest that actions that are feasible in a state of the state space are linearly constrained with 
respect to the state in the context where a state space for the MDP is a polyhedron in a Euclidian 
space", the examiner respectfully disagrees. 

Viniotis states that the state is a linear function of the control actions (Page 652, CL2, 
Para 8). One of ordinary skill in the art knows that if x is a linear function of y, then y is a linear 
function of x. Therefore, it is clear that actions are linear functions of state. Selecting an optimal 
policy (set of actions) reduces to minimizing a linear functional; this minimization is constrained, 
since the states generated by the policy have to belong to the state space, a subset of nonnegative 
integers (Page 653, CLl, Para 1). Therefore, it is obvious that the actions are constrained by the 
state, where the state space is in the Euclidean space. 

7.3 As per the applicant's argument that "the Office Action asserts that Viniotis teaches 
building a value fimction for the state using representations that facilitate the computation of 
approximately optimal actions at any given state by linear programming at Page 653, CLl, Para 
9 to Page 654, CLl, Para 4 and Page 652, CL2, Para 8; . . . Viniotis merely states . . .it can be seen 
that Viniotis teaches only the formulation of an MDP and the definition of a value function; 
however, the indicated locations in Viniotis cannot be interpreted as teaching the limitations of 
the appUcant's claim directed to "building approximations from above and fi*om below to a value 
function for the state using representations that facilitate the computation of approximately 
optimal actions at any given state by linear programming" the examiner takes the position that 
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the examiner used the above section as reference only for building a value function for the state 
using representations and facilitating the computation of approximately optimal actions at any 
given state by linear programming. 

7.4 As per the applicant's argument that "the Office Action asserts that Schneider teaches 
building a value function for the state using representations that facilitate the computation of 
approximately optimal actions at any given state by linear programming at Page 2726, CLl, Para 
3 and 4; . . . Schneider merely states . . .it can be seen that Schneider teaches only a Markov 
Decision Process . . the examiner takes the position that the examiner used the above section 
as reference only for teaching a computerized apparatus for solving stochastic control problems 
of linear systems in high dimensions comprising a computer and a logic performed by a 
computer for modeling a structured Markov Decision Process. 

7.5 As per the applicant's argument that "the Office Action states that Schneider teaches 
building one or more approximations from above and from below to a value function for the state 
using representations at Page 2722, CLl, Para 2 and Page 2724, CL2, Para 6; . . . however, the 
indicated sections in Schneider cannot be interpreted as teaching "building approximations from 
above and from below to a value function for the state using representations that facilitate the 
computation of approximately optimal actions at any given state by linear programming", the 
examiner respectfully disagrees. 
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Schneider teaches that the solution to the MDP is a value function and a method for 
generating an approximate value of this function (Page 2722, CLl, Para 2). Schneider also 
teaches that the solution to an MDP is an approximate value function (Page 2724, CL2, Para 6). 
Schneider teaches that the value function can be represented as a function of states and actions 
(Page 2725, CLl , Para 1). Trajectories through the MDP model are generated repeatedly using 
the current approximation of the value function (Page 2725, CL2, Para 4). For noisy versions, 
one could use noisy outcomes directly from the stochastic simulation (Page 2726, CLl, Para 3). 
It is inherent that when noise is introduced, the approximations to the value function will be 
determined by the amplitude of the noise and will thus be limited from above and from below. 

Conclusion 
ACTION IS FINAL 

8. Applicant's arguments with respect to claim rejections under 35 USC 103 (a) are not 
persuasive. Accordingly, THIS ACTION IS MADE FINAL. See MPEP § 706.07(a). 
Applicant is reminded of the extension of time policy as set forth in 37 CFR 1.1 36(a). 

A shortened statutory period for reply to this final action is set to expire THREE 
MONTHS from the mailing date of this action. Li the event a first reply is filed within TWO 
MONTHS of the mailing date of this final action and the advisory action is not mailed until after 
the end of the THREE-MONTH shortened statutory period, then the shortened statutory period 
will expire on the date the advisory action is mailed, and any extension fee pursuant to 37 
CFR 1.136(a) will be calculated from the mailing date of the advisory action. In no event. 
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however, will the statutory period for reply expire later than SIX MONTHS from the date of this 
final action. 

9. Any inquiry concerning this communication or earlier communications from the 
examiner should be directed to Dr. Kandasamy Thangavelu whose telephone number is 
703-305-0043. till October 27, 2004 and 571-272-3717 after October 27, 2004. The 
examiner can normally be reached on Monday through Friday fro m 8 :00 AM to 5:30 PM. 

If attempts to reach the examiner by telephone are unsuccessful, the examiner's 
supervisor, Kevin Teska, can be reached on (703) 305-9704, till October 27, 2004 and 
571-272-3716 after October 27, 2004. The fax phone number for the organization 
where this application or proceeding is assigned is 703-872-9306. 



Any inquiry of a general nature or relating to the status of this application or 
proceeding should be directed to the receptionist whose telephone number is 703-305- 



9600. 



K. Thangavelu 
Art Unit 2123 
September 8, 2004 




